03. Biases in Data Collection

Summary of the types of biases in Data Collection

Selection Bias

1. Non response bias

• Here is an interesting take on how to mitigate non response bias.

1. Voluntary bias

• Random sampling can provide strong protection against voluntary response bias.

1. Undercoverage

• This Harvard Business Review article by Kate Crawford illustrates this nicely using the example of Hurricane Sandy.

Response Bias

1. Leading Questions
2. Social Desirability
   • This post by the experience management company Qualtrics describes another example of response bias: “acquiescence response bias”.

Missing Variables

1. Features that are not included as a part of data collection that affect the analysis and final recommendation

Survivorship Bias

1. Brands that exist in collection today but their churn indicate implications on an analyses and its interpretation

Additional Resources on Overcoming Selection and Response bias

1. Simple random sampling is cited as a way to address biases during data collection. Check out this blog describing some method.
2. If you are interested in academic papers, we also recommend reading an article by the title "Addressing Selection Bias in Event Studies with General Purpose Social Media Panels" by Princeton faculty Han Zhang and Microsoft Researchers Shawndra Hill and David Rothschild.

QUIZ QUESTION::

Match the Selection Bias Example to Type Response options:

ANSWER CHOICES:

Voluntary

No bias

Undercoverage

Non-response

Undercoverage

Non-response

Selection

Selection Bias	Type
Asking a survey group their salaries as part of a feature dataset.
Polling a college group about their political preferences as a feature set to represent the larger population’s opinion.
Asking a group of older residents to take a 5 question survey via smartphone as part of a feature dataset.
Phoning entrepreneurs to ask about their financial growth and primarily getting responses from companies that are growing.
Running polls in urban areas as part of a feature dataset.

SOLUTION:

Selection Bias	Type
Polling a college group about their political preferences as a feature set to represent the larger population’s opinion.	Voluntary
Phoning entrepreneurs to ask about their financial growth and primarily getting responses from companies that are growing.	Undercoverage
Running polls in urban areas as part of a feature dataset.	Undercoverage
Asking a survey group their salaries as part of a feature dataset.	Non-response
Asking a group of older residents to take a 5 question survey via smartphone as part of a feature dataset.	Non-response
Phoning entrepreneurs to ask about their financial growth and primarily getting responses from companies that are growing.	Undercoverage
Running polls in urban areas as part of a feature dataset.	Undercoverage
Asking a survey group their salaries as part of a feature dataset.	Non-response
Asking a group of older residents to take a 5 question survey via smartphone as part of a feature dataset.	Non-response

Which of the following statements are not true?

Random sampling is a good way to reduce response bias.

Decreasing the sample size tends to reduce survey bias.

To guard against bias from undercoverage, use a convenience sample.

To guard against nonresponse bias, use a mail-in survey.

SOLUTION:

• Random sampling is a good way to reduce __response bias__.
• To guard against bias from undercoverage, use a __convenience sample__.
• To guard against __nonresponse bias__, use a mail-in survey.